{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第十周基础作业"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1、运行课程给出的mnist代码。<br/>\n",
    ">运行如下，最后测试集的准确率为0.992。\n",
    "\n",
    "2、给出代码的运行log截图并提供心得体会文档描述对整个模型构建及训练过程的理解。\n",
    ">（1）卷积神经网络的结构一般输入的数据是图像,图像是按每一个像素进行输入的，按全连接神经网络，若一个尺寸为200x200x3\n",
    "的图像,会让神经元包含200x200x3=120000个权重。而网络中肯定不止一个神经元,那么参数的数量就会快速增加，而大量的参数会\n",
    "很快导致过拟合。\n",
    "\n",
    ">（2）卷积神经网络针对输入全部是图像的情况,与常规神经网络不同,卷积神经网络的各层中的神经元是3维排列的：宽度、高度、深度\n",
    "(这里的深度指的是激活数据体的第三个维度)。（1）中数据体的维度是200x200x3。卷积层中的神经元（卷积核）将只与前一层中的一\n",
    "小块区域连接,而不是采取全连接方式。这使得前向传播函数实现起来更高效,并且大幅度降低了网络中参数的数量。\n",
    "\n",
    ">（3）一个简单的卷积神经网络是由各种层按照顺序排列组成,网络中的每个层使用一个可以微分的函数将激活数据从一层传递到另一层，\n",
    "卷积神经网络主要由卷积层，池化层和全连接层组成。通过将这些层叠加起来,就可以构建一个完整的卷积神经网络。比如mnist图像数据\n",
    "分类的卷积神经网络的结构可以是输入层-卷积层-ReLU层-池化层-全连接层:\n",
    ">>1）输入28x28x3存有图像的原始像素值,本例中图像宽高均为28，有3个颜色通道。<br/>\n",
    ">>2）卷积层中,神经元与输入层中的一个局部区域相连,每个神经元都计算自己与输入层相连的小区域与自己权重\n",
    "的内积。卷积层会计算所有神经元的输出。如果我们使用32个滤波器(卷积核),得到的输出数据体的维度就是28x28x32。<br/>\n",
    ">>3）ReLU层将会逐个元素地进行激活函数操作,比如使用以0为阈值的max(0,x)作为激活函数。该层对数据尺寸没有改变，还是28x28x32。<br/>\n",
    ">>4）池化层在在空间维度(宽度和高度)上进行降采样，数据尺寸变为14x14x32。<br/>\n",
    ">>5）全连接层将会计算分类评分,数据尺寸变为1x1x10，其中10个数字对应的就是mnist中10个类别的分类。全连接层与常规神经网络一样,\n",
    "其中每个神经元都与前一层中所有神经元相连接。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 卷积神经网络实现Mnist手写体识别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:50.515650Z",
     "start_time": "2018-06-01T06:32:43.133728Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "os.environ['CUDA_VISIBLE_DEVICES']='2,3'\n",
    "\n",
    "# 将TensorFlow日志信息输出到屏幕\n",
    "# TensorFlow有五个不同级别的日志信息，分别为调试DEBUG<信息INFO<警告WARN<错误ERROR<致命FATAL\n",
    "tf.logging.set_verbosity(tf.logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T05:39:48.304594Z",
     "start_time": "2018-06-01T04:55:17.674707Z"
    }
   },
   "source": [
    "分别看一下训练集、验证集、测试集数据的形状"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.075054Z",
     "start_time": "2018-06-01T06:32:50.518290Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /home/silence/DeepLearning_Dataset/MNIST_datasets/train-images-idx3-ubyte.gz\n",
      "Extracting /home/silence/DeepLearning_Dataset/MNIST_datasets/train-labels-idx1-ubyte.gz\n",
      "Extracting /home/silence/DeepLearning_Dataset/MNIST_datasets/t10k-images-idx3-ubyte.gz\n",
      "Extracting /home/silence/DeepLearning_Dataset/MNIST_datasets/t10k-labels-idx1-ubyte.gz\n",
      "#########################################\n",
      "Train_data_images_shape: (55000, 784)\n",
      "Train_data_labels_shape: (55000, 10)\n",
      "#########################################\n",
      "Validation_data_images_shape: (5000, 784)\n",
      "Validation_data_labels_shape: (5000, 10)\n",
      "#########################################\n",
      "Test_data_images_shape: (10000, 784)\n",
      "Test_data_labels_shape: (10000, 10)\n",
      "#########################################\n"
     ]
    }
   ],
   "source": [
    "filename = '/home/silence/DeepLearning_Dataset/MNIST_datasets'\n",
    "mnist = input_data.read_data_sets(filename, one_hot=True)\n",
    "\n",
    "print('#'*41)\n",
    "print('Train_data_images_shape:',mnist.train.images.shape)\n",
    "print('Train_data_labels_shape:',mnist.train.labels.shape)\n",
    "\n",
    "print('#'*41)\n",
    "print('Validation_data_images_shape:',mnist.validation.images.shape)\n",
    "print('Validation_data_labels_shape:',mnist.validation.labels.shape)\n",
    "\n",
    "print('#'*41)\n",
    "print('Test_data_images_shape:',mnist.test.images.shape)\n",
    "print('Test_data_labels_shape:',mnist.test.labels.shape)\n",
    "print('#'*41)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T05:49:40.128071Z",
     "start_time": "2018-06-01T05:49:40.123888Z"
    }
   },
   "source": [
    "可以看出总共有70000个样本，其中<br/>\n",
    "训练集：55000个样本，images里面每张图片是28x28长度的一个一维向量（所以用的时候需要先给它还原成28x28的二维图片）<br/>\n",
    "&emsp;&emsp;&emsp;&emsp;labels中则是图片对应的数字的值。<br/>\n",
    "验证集：5000个样本。<br/>\n",
    "测试集：10000个样本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.695746Z",
     "start_time": "2018-06-01T06:32:51.077167Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcoAAAHRCAYAAADqjfmEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3dd5RURdrA4bcmEGfIipLjkJSwijmggAEx54S6KAomVkVdXT+XddVdIyAmwIAR06JgwggmFEQUQXKQDJIzTKjvjxmqbrXTRU9P9/R0z+85Z8++1VV9u+ROd91bdatKaa0FAAAULy3RFQAAoDyjoQQAwIOGEgAADxpKAAA8aCgBAPCgoQQAwIOGEgAAj6RuKJVSWim1XSl1f4Tl+yqlthW9r1W864eS4XymnijO6eCi8loplRHv+qFkKup3VCXzggNKKS0irbXWC4rSx4rIRyHFqovIeVrrd8K9D+VDMeeznoi8JyJtRSRdRGaLyG1a629970P54Ts3Sqk+IjJaRK7RWo8KvN5MRBaLSKbWOq+MqooIFHc+lVLpIjJYRP4qItkiskBETtBab/K9L5mk1BWb1vprEcnam1ZKdROR8SLycaLqhFLZJoVfvvkiokXkTBEZr5Tanx/Q5KaUqi0id4nIrETXBaU2WESOEpEjRWSpiHQQkV0JrVGMJXXXawSuEJG3tdbbE10RlJzWepfWeq7WukBElIjki0htEamT2JohBh4UkWEisi7RFUH0ii54Bkphr8DvutBMrTUNZTJQSlUXkfOksGsHSUwpNUMKr1DHicgorfXaBFcJpaCUOkxEDhWRZxJdF5TawSKSJyLnKaVWK6XmKaWuT3SlYi2lul5DnCOFV6uTEl0RlI7WuqNSqoqInC0ilRJdH0SvaDzrKRG5QWtdoJRKdJVQOo1EpKaI5IhIcxFpLSKfK6Xmaa0/TWjNYihl7yilsNv1JZ3MTyvBKOqGfV1E7lRKdUp0fRC1ASIyQ2v9faIrgpjYWfT//9Ja79RazxCRMSLSK4F1irmUbCiVUo1FpJuIvJTgqiD2MkWkRaIrgah1F5Gzi7rpVkvhQyCPKqWGJ7heiM6Mov8P3pCk3M1Jqna9Xi4i32mtFya6IoieUuoIKfwbnSKF00NuEpH6IvJDIuuFUrlSRKoE0v8TkbdF5LmE1AalorVeqJT6WkTuVkrdJIUXsReJyMWJrVlspWpD2UdEHk50JVBqlaXwycgWIpIrIr+KyGla65UJrRWiFpxbJyKilNojIlu01psTVCWU3sVSeKGzXkTWisg9WuvPE1ul2Er2rtfdIjJNKXVf8EWtdVut9Z+uUJVSVymlNhW9r6CM6ojIOedTaz1Ja91Ja52tta6jtT5ea/3V3sKcz6RQ7Hd0L611t5DFBu4VkV+K3pdyXXgp4E/nU2u9Qmt9itY6S2vdQmv97N68VPmOJvXKPAAAxFuy31ECABBXNJQAAHjQUAIA4OF96rVn2vkMYCbIpwVvxWXJEs5p4sTjnHI+E4fvaOoJd065owQAwIOGEgAADxpKAAA8aCgBAPCgoQQAwIOGEgAADxpKAAA8aCgBAPBI1W22AACJlpZuwnkjuzhZs05+ysSnX9HfxBmfT4t/vUqIO0oAADxoKAEA8KChBADAgzFKAEBMZDRt7KTnPVjXxIu7jQopXclEm1rauN7ncalaqXBHCQCABw0lAAAedL2iXEtvn2PiOf1rO3nzz3naxAXibuGXJnZbuac2NTfx6Md6OeXqPjc5JvUEKqqMFs1M/Nvd9Zy8P3e3WtcsO9rE9b9eZ+L82FUtZrijBADAg4YSAAAPul6RcBmNGznp3+49wMSvn/isibtULnDKFQSu8wrEzQteA/artcDEDe541Sn1/IRjTZy3fEXklYYjrUoVEzf5Sjl5TzX81sTpyp6X2Xt2OOVuPbmPifPnLhCUXyrTPqU6+591TLy4R/iu1haf/dVJt+n3m4kLds2PYe1ijztKAAA8aCgBAPCgoQQAwCPpxihX3XKUk1aBWQFV1tvExrbu+w6cbB86rjJ+SlzqhsgteuhIE8+59EknLzjVIzjNoyDkuu6DHTVNPGVbi7CfdUj1JSY+N2uLk7dywkwTv9/BnX4Cv+C45IoxdgrO+w1fLa64iIh0m3mWidWj7lSCygt/LnWdMpo1MXHekqWlPh6KN3d4JxMv7jEybLlWE680ces+Pzl5oU8VlGfcUQIA4EFDCQCAR0y6Xtde73aHbuqYa+KxJw2PxUcY7SpNDZu3S+eZuGZaVSdv7eXbTbxymPuf/djqniZef0ENE+ctWx51PeF3fk87ZSB0VR13qoe9lntyU0un3KcndzCxb2rHt6dfZOIznnnayQtOHXlfuvorDceCf9qNeOd0fTJsudafX23iNv3nmrhg+xKnnPtXEJl5I9xz9t5JT5j4whdvcfKa/PO7KD4BIiILhhzhpns/FUjZ72iLT90pIDn9Zpk4mvNbXnBHCQCABw0lAAAeNJQAAHhEPUY5b6QdG5jTa6iTV1llBlPRfkSJuZ/r2j+9eiB2815q+pWJL3ujm4k3XtLEKcfj5qV02MEmvK6uHSv8YMcBTrHgVI+ZWxqYePeg/ZxyCx+yJzLnvmpOXv5suyRWcDpQ5rPuyc8NDJysuMMda2/4X8a0gvSRnZz0V5c8HEjZf/+lee7SdDl97RScgtw9pa5Hbo9DTDy2p/sMRIfA0moonT2n2N/4sWcNcfLSlZ0a5EwBueoXp5wuKI97gZQcd5QAAHjQUAIA4BF11+vTJ7xk4tAuz/+ub23itXuyozr+/6bZ7pUm45WnZGSWd7fXBA/1es3JC67W8kqziSa+7LVuTrmNF9pdLpg6EoUpv5qw37n9TZy+aoNTzJ3qsdpEK+5wV9+ZfbydCnDqyGucvPTZNl7f164ClKunOeWCU1Gavvq7W4/Q+ldwa+5wu033T7fdrTu1zesz8FanXLXcH2Jaj21/s9/Xgyu5vz3b9G4TN39rvZOXGp2AZafu3YtN3LFSFSev5+zTTZxzrz0f+SnS1RqKO0oAADxoKAEA8Ii663XIheeZ+B+dazh5+79rV9/IX+92q0UqR8KvwBONVuNtPOr5Xk7e6jF2lZjray0zcbAbVkSkTT/bXdjsHrpeS0NPtd2wkXZxVlnnru0xYnMzE1das83JWzTYPsH64uW2iza4yLqIyLTd9lqRjZv9+uV8Ezbv7Lnnm7ja2PBdrSrD/uSoqlXDlguVf7Dtdn+83Qthy3WbdpWJ9581J+Lj489uavhZ2Lwto+0wVK35k8uiOgnFHSUAAB40lAAAeNBQAgDgEfUYpZ5mV4Wv6z5xX+4fwy6Y4Y5dvPB4bxNfP/jp0OLGa5fZFYjuuuew2Fesgtp5pvtvuaGt/bMMjkvW/dUdh+xXc4mJO7/vTu04rLJ9X3AKyNTd7rXhP/raaSXp4m4si8hlZ+4y8faQvNyTDjVxnXuWmPiNFp+U4BMmFfvqtyHnc7//lN1KYKlo82V2l5DjqtiNtI+ecY5TrtbL35dZncoD7igBAPCgoQQAwCMmGzcDpbHyQnfFl9nH2+7v4HSO0A2eg3nBrtbQvOAUkMvfvsEp1+LL1H+0PVZGjDzdSV93m12Q/KUWdv7Vdd+d4pR7rqk9nxkSsiNBKV05/jon3XpyxeoSjLVNZ4R2nBfaMc7duCBLL4pfJdJC/kbKwWo/3FECAOBBQwkAgEeF7Hpdfpe772BBl60Rva9+uu0izDvxECcv44tpocURpeBTqsFrOfd1f16/ZSeaeNnf7SL9dLVGb3uj0H9/q6qy+0CObvpFSK7tSrt1tX3C+cMJXZ1SuQfa79eCk0ZGVKd6P5V+wwRYB9beUuzrVdeHP/fR2n2qPf/rrrF7mB5Uf5VTbut59m8rb9VqSQTuKAEA8KChBADAg4YSAACPpBujzGjRzEkv6HugiZ+6aEREx+hWxV2BJV1Fdr3QKCPLxCNeGOrkDWh6TETHwJ81eKOSkz6/oZ2GcFCNlSa+ru53TrmGgY2DQ6/5Fj7YzsRVv5wSg1oi59k/nHS73Osjel+rl+0OQgVzF5q4eZ47XrzoP0dKJAasONrEdV5znw3QoYXhlXFAfSc9ss2rgVSWlFZ6rZomPmvyfCfvwuxhJq6ZFn4nmQ7DLzVxo3MZowQAoNyhoQQAwKPcdr1uO/9wE//xF9ue/+ucMU65i7I3RnH00l8f9PhsoJPOkR9LfcyKqup7btfo7vdsPC1wrvp17e+U23qfXUXki4PfcPKO+addoeWXaY1NzObM0cuft9BJN79zYZiSIe+L8PgZOyKb6vHjqM4mrpfLdJ9Sycx0kk0yStfdunaAO/XurGsnmrhfzZUhpSPbuHu/7OJXCypL3FECAOBBQwkAgAcNJQAAHgkdo1RdOpi41nB32aIPm9kdByKdvvHudtu/PnNno7Dl3n+om5NO320fKr/iX3YXhD/3qVuVVmeGzatIMhq7/855y5bH7bP01F+ddFZgk4rzJ7k7W4xt9aGJD7raTt1p8k/GKMsr5RnMzAuMdNaet7sMalMx6K3u8p0jNjcwse/3L71eXRMv+2sbE/868KkY1q7Q5p1VTLx/zI8eGe4oAQDwoKEEAMCjTLtefx/sPjp8z0X2kf5Ls9c7eUvz7Gryc/bUNvGNr1/tlKu2yj5SfuDEdSbO/21e2HrUlPCbu87/e2ClipCuh8W520zc7L1tUlHtPNPuABGchiEi8v7vtjv9wLNml1mdNj/SxEkXPGO703Nb7yyzeiB6V108IWze+Qts13r6xJ/ClkPJ5G/a7KRfX2539OhX087TOvqOH5xyXe+zGzdfkPV5TOs0+I/2TrrBTXZ6SF5MPyly3FECAOBBQwkAgEeZdr3W6rrWSQe7W7v/doaTl/vEASYOrtzSTMKvxBHpCiChCo7vYuKzaj0XyHGvIzYUBBbvnuI+gZnqgk+3XvjgRyb+cUszp1xZdrcGF1w+7z9ut12asKFveZe+335OunXlBWHLrnu6mYmzJTELY1cEu16wm0zsfjjXxA8fMD3mn5Wr7S92+0l9TZzzd3cYLu/3ZTH/7JLijhIAAA8aSgAAPGgoAQDwKNMxyrp93SkVrW6xu0G0HOSOPWbI0jKpk4jIxhy78sPRVcJfO/SbeZmJ60n46Sep6PdL7PSL4GPjj0/v4ZRrKbEfyzAOO9hJnvrCV7ZOtdzxrYLANWDmvMh2KUDZ2nxCSyd9ejU7zrxNu6vvVFmXK4i/Gq/Z6V4//NuuPnZcleJK71u+LjDxoT9e4uRVettO+2vxsv39T9QUEB/uKAEA8KChBADAo0y7XvNWuY91txxUPh7zXt+1+Jv92Xt2OOnsp2oWW64iaPilXTw58+Z0E9/c+Qun3HM3nmbiurPc7rOML6YVe+z09jlOemX3eibOOs3+jXx58ItOueAUkIKQa76cj6618eDviv1cJNYVg8eFzVuc657PzM+K/9tBYrT95nITq5nZJm4+bJZTTufbrtf9t86Jf8XihDtKAAA8aCgBAPCgoQQAwCOhGzcnyskztzjpsbWeDKTsMnVXzLrCKVf7o6nxrFb5Fliy7+gZ55j4i4PfcIpdd+cTJi6QAidv8NpDij30GTVfd9JdKtv3pQWu5UKPF7zOa/P29U5O+4ftslfl8XFziNRND78DzyOrTg55ZVN8KwOv9k8PcNLNHrTLiuo8+w2LdhnR8o47SgAAPGgoAQDwqJBdr+fVmOGkq6VlmXhert0ktNrwWmVWp2RS65o9Jh48zu1OfaC+/bfN1U6W3Lf/zyYuEJsZutNHcKrHmny76fJT692Nvz8ZfrSJWz/nruxEd2ty21OQvu9CiKv7W3Q2cWNxp1jp0MIpjjtKAAA8aCgBAPCoMF2vawfYbrv66e7Tq4tz7dN3Fz8wyMT1Pgq/SXRFlrdsuYl/Ob2xk9fqv8U/2SoiMrvbKBMfN+MCE/+xoUbY97QaYjtR9VR3s+y6nk28kdxGNnvfSR/y6N9M3PLW70OLA3HFHSUAAB40lAAAeNBQAgDgkbJjlKpyZSd97nV2l4utBXucvF5T7AbSTZ5l3Ksk8pavcNItL10RpqRIb7HjlzVkYSAOr6I9hl6R3D3mUifdts9jNs50v79S4E4hAsoSd5QAAHjQUAIA4JGyXa9S4HbavTz+BBN/9Es3J6/JmzxuDpS1pv/nDnPc8n9Hhi3bkqlASCDuKAEA8KChBADAg4YSAACPlB2j1LnuFJBmdzPGAQAoOe4oAQDwoKEEAMBDac3aJwAAhMMdJQAAHjSUAAB40FACAOBBQwkAgEdSN5RKKa2U2q6Uuj/C8n2VUtuK3tcq3vVDyURxPnsUnc8CpVSPeNcPJcd3NLVEcT4HF5XXSqmknbef1A1lkU5a67v3JpRSI5RSc4t+PK8MFtRaP6e1zirzGqIkQs/niUqpn5RSW5RSi5RS/fbmaa0/KzqfSxNSU0SK72hqCT2fnZVS05RSO4r+v/PePK31vSLSISG1jKFUaChD/SIiA0Tkp0RXBKWjlMoUkbEi8qyI1BSRC0XkMaVUp4RWDKXFdzRFKKUqich7IvKKiNQWkdEi8l7R6ykj5RpKrfWTWuvPRWRXouuCUqsjIjVE5GVdaKqIzBaR9omtFkqD72hK6SaFS6EO0Vrv1loPExElIicmtFYxlnINJVKH1nqNiLwuIlcppdKVUkeKSFMR+SaxNQNQpIOIzNDuyjUzJAW6W4OSdnAVFcbrIjJKRIYWpftrrZclsD4ArCwR2Rzy2mYRyU5AXeKGO0qUW0qptiIyRkT6iEglKbxKvV0pdVpCKwZgr21SODwSVENEtiagLnFDQ4ny7CARmae1nqC1LtBazxWRD0Tk1ATXC0ChWSLSUSmlAq91LHo9ZaRcQ6mUqqSUqiKFA8qZSqkqSqmU+++sIKaLSOuiKSJKKdVSRHpL4RgIkhTf0ZQyUUTyReQmpVRlpdQNRa9/kbgqxV4q/nF+IiI7ReQoERlRFB+X0BohKlrrhSLyVxEZJiJbRGSSiLwjhWOWSF58R1OE1nqPiJwlhcMjm6Tw+3pW0espI9kbyt0iMk0pdd/eF7TW3bTWKuR/E0VElFJXKaU2Fb2vIDFVhkdx5/NNrfVBWutsrXUjrfUdWusCERGlVPei81lfCq9qUf7wHU0txZ3P6VrrQ7TWVbXWf9FaT9+bp5S6Vwrnze4WkaTd05H9KAEA8Ej2O0oAAOKKhhIAAA/vggM9086nXzZBPi14S+27VMlxThMnHueU85k4fEdTT7hzyh0lAAAeNJQAAHjQUAIA4EFDCQCABw0lAAAeNJQAAHjQUAIA4EFDCQCABw0lAAAeNJQAAHjQUAIA4EFDCQCABw0lAAAe3t1DktniMR2d9DdHP23iS/rc6OSlf/lTmdQJQHgLHz3CxDef8pGT9+HFR5q4YMacMqsTInCE/a1dfLO7+ca840ebuNXEK528lpf8HNdqxRJ3lAAAeNBQAgDgkbJdr3ppdSdd99iqJt7QprKTt9+XZVIlxNju07qaeMM125y86V1fjegY1y0/1sTffNTJyWvx7CIT561aHU0V4ZHRsIGTHn7mCybuWXWnkzf68F4mrjsjvvXCvq0eeJSJH7jheROfVHW7Uy43sAX10MPGOHnDpG2xx15z41FOusFrtqs9f/2GEtc1FrijBADAg4YSAACPlO16rb5chc074MLfnXT+M/GuDaKlMiuZeN5jXZy8D05/3MStMt3u9IIIj/9Mo6/te675ysnrfHAfEzc6l67XWFt4bVMnHdrdisRSle13auMFf3HyvrrtURNXU5WktJb/3Xa3Tr1+iJP35vWNTDxsyLlO3n7PTC71Z0eCO0oAADxoKAEA8KChBADAI2XHKH125mU66dL3sCNe5j7R2cTzTn/KyUuTKiYuEC2R6Lesm5Me1XhS2LLDOtvH2R+te7yJE/WIeqppfPTyRFcBHov+acclZ/UZHpIb2a/mM5tamPjZl09z8hrKdybeXdc+VZCp0p1yl2avMnHXOx9z8i6XW0wcz/FK7igBAPCgoQQAwCNlu15rnLYqbN7md9wVQfaT38OURFkITgERcbtbZ/UOdvm4XTKr8neY+Lixtzl5LcbuMXHl+XZqR/669U65Lm9cauJpXV9x8n7a2czEek9umNqjJHb1PszEQ1s8EZKbKUis4JSQ6u03lvj9H+3IdtLv3H6SiRt+8F1o8RLLCfmtGPP3R0x8cpeBtty1U0v9WUHcUQIA4EFDCQCABw0lAAAeKTVGmd/NPs48vsOTTt7Pe+z4Vv1XZzp5kS53hvhYdf2hTnre6cGxK3ventvcxCn3v2t6mrj1t9+HPX6e57N37w4/LjZ+hd2QturWxZ6jIFI769rzeXAlxiQTTWW4TcDCf9nf0N8ODZ0SUrzglKu157pjlJVXRDZW2OwD+0xBx6ZXOnnTjnzOxKFTR5pn2CliNebE7++JO0oAADxoKAEA8EitrtfKtt3PUu5uErnartxSsHVrmdUJ+9a/33tOOk3szi8Prm9v4sln5Djl1JKfIzp+eo0aJl5+9UFO3u0d/2fi6XvcTviqJ9Pdmijf7nav4bOX+TrQEa3dPdwdeX67LLLu1ptXHm3iNafZLs/89Sujqkf6lz+ZuMmXbt7YuQea+IKstVEdv7S4owQAwIOGEgAAj5Tqel1yNu1+MsoPuV4LLnD+4QPdTJy9JPyTrZLmPg2Xf3wnE/ce/rmJr6vl9usEu3lPm3tWyEFXhP88RKXtdbMiKjdkeU8nXenj2K60UpGtuclukjyg/7sRvSfY1Soisvh4+50t2JH6mwTQsgAA4EFDCQCABw0lAAAeKTVGmX0A0z5STbXVe/ZdSNwxSRGRj14ZGdH7zl7Qy8Rp5+5w8vIjOgJKYkD94BixCltu7ketnXQj+SNONUp9aZ3aOen/3GRXuuledUdocSO44k5wCohIfMclVZcOTrpZ5k9hSoosyN1t4pqL4jeFiDtKAAA8aCgBAPBIqa5XJKf5O+u7L9RcYsLnXxpm4v+s6eEUm/h7KxN/fNgwcVU10eaCXSbu+sHfnFJtb7XTFQq2b4+0yoizpu+6Xa10g0fv2Jfdrktfd2vQ1HcPNnHD9aXfdDlSc/tXc9KHVdZhSopM2G5X7qr63pS41Yk7SgAAPGgoAQDwSPqu17Qqdj+yYxqGX8R65NrjA6ltcawRSmr29e3dF975wYQHptsu1KENvnWKpTWw3UEFga7WUCc8McjEOQ+5XUjsRRp/wZVg2mQG//2rOOVW5Ae6BPPobC2NddceaeL+tR8NybUbRqzK3+nk3PK7XZ2qyf/WmDjeZyOjeVMTTzrl8ZDc8N/tbza0CqTWxbZSAdxRAgDgQUMJAIAHDSUAAB7JP0ZZq6aJn2jwUdhyk76xG/a2FM8uFCgTu0/rauJlF7kraqR5VmwJSleB6zztjjZ2n3WOiRs8VHaPtkMkvf7+TrrLJb+auEZaldDiRrext5m49Xy+o6Wx1Q75SVZa5bDlHll7gvu+Y4PjfPEb8ws193q7OXPwuYRQGwNTvUREVg9taeLqjFECAJAYNJQAAHgkfddrXrP6+y4kIk0+zo1zTRAqrWNbJ33ACLsR8qjGz5o4uFFzYbp4d67u6qT/N+VQEz/dc7ST91ybV0zc5wLbpZf1Jl16cVevtpMc1fjjYottCelGy17MdXtZ+/izQ510c5lcdh+u7BCLTveUC7ht+alOuvrbP4QpGVv8ZQIA4EFDCQCABw0lAAAeST9Gue7uXcW+3mvOGU660sRfTBx+LXqU1rp+dumsCfc84uTVdKYGhJ8CcuuqI0z80Rd2DCXncXeJwpxVdreAR0641MkLbtx80b122tD7b7rjZ4i9/OqVIir3a667S8QBQ5jGU9YO/DZxSwVuvvRwE8+54MmI3vPdt+5yl2U11Y87SgAAPGgoAQDwSPqu16cPejWQss8Yr9xSwynXIG95GdWoYtl60RFOOtjdWjNkFZbZuXaKzuOre5p47pAOTrma7/5s4ha77OPq7vo9rvRJvzjptm9eb+Jfzh9i4rEn3eCUy/zkR89REY3sR1dFVK7/dLe7vJHMClMS8dL0rjlOes342B4/o1FDE8+/vomT98NlwV1Nwq8e9PpWOwUw54WNTl5ZdRxzRwkAgAcNJQAAHknX9ZrRzL19z1b2Sbl0lVnW1anw1nV0n14NdreO3V7HyXvhgtNMXPDzbybODnlyLZrNlNOqut28Hf6yxMSVA38XBRmRLbiOkslo3MjEOVlLw5a7dEkPEze9eqWTx1bNZe+YWguc9Lut7VBK/vxFER0jvV1rE8+/op6TN+S8F0x8UtXtIe8M390aNPr6M02cMWtaRO+JNe4oAQDwoKEEAMCDhhIAAI+kG6PcNcpN52Tasan8wOa9WW+600NQNoKbLt/x5QVOXs7PU2P6Wen16pq42lh37PGNFh8GUoxLxtvqXo1NPG7/cU5ecIPtjbvsajxpe9xH/VWmXdFH5+6JdRUrlNaj7BSdwb06O3n37menX11VY5mTlz7O/ob+uqORRKJz9UkmvjQ7sqlBocZttytm3fbZRU5e2+/ttKFonl+IBe4oAQDwoKEEAMAjKbpe03NamvjWZuPClrt4sV3tpcaYstnQs6KrN8NdYn5jwU4TT+01xMnr+uxAE7f7v99NnL9mbdjjZzRsYOLtnRo6eQOHvm7i06ptdvKCXTRPbrJ/P1W/nhO2HOIjOCTyYdvA93eeW6712wNsfDMbbJdG3qIlJp4w7Bgnb+Bg+28bunpWnxp2c3UJxjGwQ7vd6U9usF3CX/3Vbsqe8+MUp1x5+I5yRwkAgAcNJQAAHjSUAAB4JMUY5Z6GNU3cverusOXmvdHGxPU1m8CWhewx7ljSca0GmfiX/k84efN6P2PiWSfZvUAGzr8w7PFfbWd3hwkdTwlORQkdxwhu/jznRrvZq9r6iyD2qmywZ2Bh3k4nr2VG1WLfszNkzKraKq7b46HO85Od9P/1727i6/ab6OS1y4ztMqDB5wNeHnqqk1dvRLBeM2P6ubHGXyYAAB40lAAAeCRF16vPdcuPNXGD1+eamJ0IEqPOHPsv/8ymFk5e+yp28+xuVU5zXdwAACAASURBVGy36acd3vEcsUrYnGc2NzXx4x/0dvJa3zPdxGoX3a3xlvWWnY51wQGDnLyf//6Uif+9rq2J3xlxolOu4XCGS8rCwq67THxnq4vdvCsPMPHJp9hNzR890B1i6fCS3QBdeX5sW7623sT1fpscvmA5xx0lAAAeNJQAAHgorXXYzJ5p54fPRFx9WvBWXFbyTuQ5DW66Pf8/tcKWe/Av75r4u62tTDx+wuFOueZ3JVdXTjzOKd/RxEnF72hFF+6cckcJAIAHDSUAAB40lAAAeCT99BAkj7wlS03c/KKlYcuNkOC0ErviS3NJrjFJAKmBO0oAADxoKAEA8KChBADAg4YSAAAPGkoAADxoKAEA8KChBADAg4YSAAAPGkoAADy8u4cAAFDRcUcJAIAHDSUAAB40lAAAeCR1Q6mU0kqp7Uqp+yMs31cpta3ofa3iXT+UTBTns0fR+SxQSvWId/1QclGc08FF5bVSit2NypmK+pub1A1lkU5a67v3JpRSpyulZhadnO+UUu335mmtn9NaZyWmmohQ6Pk8USn1k1Jqi1JqkVKq3948rfVnRecz/J5dKA9Cz2lnpdQ0pdSOov/vvDdPa32viHRISC0RKXM+lVL1lFLfKqXWK6U2KaUmK6WO3lswVX5zU6GhNJRSrUXkVRG5TkRqich4ERnHlWlyUkplishYEXlWRGqKyIUi8phSqlNCK4aoKaUqich7IvKKiNQWkdEi8l7R60g+20TkryKynxSez/+KyPhU+81NqYZSRE4Wka+11t9orfOk8KQ1FJHjE1stRKmOiNQQkZd1oakiMltE2vvfhnKsmxRuGD9Ea71baz1MRJSInJjQWiEqWutdWuu5WusCKTyP+VLYYNZJbM1iK9UaSpHCkxWMlYgclKC6oBS01mtE5HURuUopla6UOlJEmorIN4mtGUqhg4jM0O4E7hlCd2tSU0rNEJFdIjJOREZprdcmuEoxlWoN5WcicrxSqltRV85dIlJJRKoltloohddF5P9EZLeIfC0id2utlyW2SiiFLBHZHPLaZhHJTkBdECNa645S2PtziaTghWxKNZRa6zkicoWIDBeRVSJST0R+E5HliawXoqOUaisiY0SkjxRe8HQQkduVUqcltGIojW1S+IMaVENEtiagLoihom7Y10XkzlR7jiClGkoREa3121rrg7TWdUXkXhFpJiJTE1srROkgEZmntZ6gtS7QWs8VkQ9E5NQE1wvRmyUiHZVSwSGSjkWvIzVkikiLRFcillKuoVRKHVI0nrWfiIwQkXFFd5pIPtNFpHXRFBGllGopIr2lcEwLyWmiFD7wcZNSqrJS6oai179IXJUQLaXUEUqpY5RSlZRSVZVSd4hIfRH5IdF1i6WUayhFZKiIbBKRuSKyUUSuSWx1EC2t9UIpfPR8mIhsEZFJIvKOiIxKZL0QPa31HhE5Swq70zdJ4fk9q+h1JJ/KIvKkiKwXkRUi0ktETtNar0xorWIsqXcPUUrtksKHPIZpre+JoPxVIvK4iFQRkfZa60VxriJKIIrz2V0KG87KItJLa/1lnKuIEorinN4rIrdI4TmtrrXOj3MVUQIV9Tc3qRtKAADiLRW7XgEAiBkaSgAAPGgoAQDw8C5c2zPtfAYwE+TTgrfUvkuVHOc0ceJxTjmficN3NPWEO6fcUQIA4EFDCQCABw0lAAAeNJQAAHjQUAIA4EFDCQCABw0lAAAeNJQAAHjQUAIA4EFDCQCABw0lAAAeNJQAAHh4F0UHEiGjaWMTbzq8oYlX9d7jlOv/l0kmHlh7npN30DdXmbhgSXUTtxr8i1OuYMeO8PU48AAT561ava9qAyklr/shJl7fobKTt3N/u267brXdxHd0+sQp17em/d58vMM9xqARfU3c4KHvSlfZOOOOEgAADxpKAAA86HpFwq0cdJSTvvvq1018dtbasO9LC1znFUiBkzfjmOds4hgbdtp1s1Ou6b3hu3wqv5Fv4rzjwhaDiIiy2/it7X+kk9X/xndN3K/myqgOP2JzAxO/e8YRJi5Ystwpp3Pd7nmUzObL7L/tF/8ZZuLKym0qCqT4LTPTxN3OMVfbct2rusMc39z0qImPSr/VxI0eLH/dsNxRAgDgQUMJAIBH0ne9pnVqZ+K5t1Q18eWdf3DK3Vhniom7PzrIyTtgSPm71U916e1zTBzsahUJ3936R/5uJ/17XjUT50umk3doJdsFlx7oFvzl6qFOua5bbFfsgY+6fwfH1Flo4glSo9g6VWhp6SZcdvfhJv71uuFh37Jb2+7slXnu+awS6LXbP72ak9e3hu1i7TvxbRMP3djKKfd574NMnLdkadh6oHhbztpm4kxlz29oV+vSvJ0mvnv5GWGP98OcFvZ41d1u8W+OftrER51ln0Zf9pj7dKze7f6dJAJ3lAAAeNBQAgDgQUMJAIBHUoxRqsq2z3p1v0OcvB/utGNOWwtsH/gRY25zyn3V2Y5lHH/ZVCdv7pCYVBMlMOfOLBOHjkkGz+MJP15j4vpDqzjl0if+FPb46661UxR6D/jKxHfV+9kpl+8Ohzi+2dAykPojfMEKasWgSMcl80zc6TU7Jtzi9slOufR2rU085+/ZTt7ME58xcXCqws21F7gf9r4NP+vW3MnKX7c+bB1RqNk1K0w84GM7J2rmhgOccrUDs6zy5y2UcHJkQ9i8w5/5m4nnnW7HKzvfeqNTrtEDiX+GhDtKAAA8aCgBAPAot12vaVVsN9ucIR1NvOB0t4vniU22u+atwaeYuOWbId06ObYbbUbLzk6ePt0+l56xwz6+nvH5tJJWGxH637FPB1Lu9dqA3+3j5g3O/i2q49d71p7/L9bapXnuGv5zccWLNfdj+7fViK5XURnuz0WloyPryjzof7YrrXVId2tQ/uz5tlwfN+/Yfrav76E7Rpi4W5Vcp1ywK/bz7IPdg9D1uk/5GzeaePpIO3xRa6E7RSN/Xvhhj0ilby/+Pq1Dr7lOevMDpf6oUuOOEgAADxpKAAA8aCgBAPAoN2OUadXcJatWvNbUxAu62kfDH9vY2ik34cbjTZz15fdhjx98hLnaxi1O3sDJE008arV9JHrz5/uoNKJ2cCW75Fzo8lhT59nH+nOk9ONK2TPt+OI3u9wpJnVn5YUWN7QKm1UhpTdp5KSnHvJ6seWe2NTCSbd9xo575YcWjlC9EXZsc+w1h5q4W4PwY54onbqjEvNv27ueu7n6q9IoTMmywx0lAAAeNJQAAHgktOs12N0659GDnLxgd+sjG9qY+Ksz2jvl0heX/DHlZVe63bfdq04w8Yb97PFeqtXRKZe/aXOJPwvFO2HmuSb+9KA3nbzR3UaZ+H5xp/JEKq+7XcFpv/tst3uLDPcc1rt1sYm3v+ceQxW/N22FteTCBmHztmk7fWDMA6c4eTV/Cz8kEo1FVzYz8bfj3V2Cjq5sN/Ce38+tb4t77KozOi98lzviY/epXZ30lT0nFlvu3bVdQl5J/NQs7igBAPCgoQQAwCOhXa9/XNrJxAvOeNLJ+2CHXTT7qzM7mDhv8ZJSf+6emuH71Gbvst01dLXGT9ZA+6f39NtuV3i/mvNMPO+pw0zc/r+rnHJrTrJPw51+wyQnr08tu1h+g4zgyufuKugvtRhv4t693MWY86rS95pet46J77jizbDl3t5qn1Su+Wpsu1pD5c+yK7dcMaGfk7fgDDtkM7uP+5ty2juB5X5+nBmfylVw6TXcDc7XXGx/u68d6I5tBDfjXhLYCHr9w+5i9lXoegUAoHyjoQQAwIOGEgAAjzIdo8xo6D6uffug10y8In+Hk/fgvQNMXGNR6cc8Mlo0M3HvU38IXxBlIrhTxMtDT3Xy+t9r8+acGRhnOtM9RlrgOq9ACtxMKX5H5jtWH+mkx39lV3lp++tyJ+/ah+zOJRPuccdeKgoV2MXn0uy1npKJUWNOyE/YGcWXExGZe539b8m5Ok4VSlFpnd1peSu71TLxljZ2qs01R7vPCgyq+6XnqHbpqx4f3mLinPFToqxl/HBHCQCABw0lAAAeZdr1WlDX7b46t7pdLPlf6w538mq8VvLu1uDGsisGHubk3XnNGya+KCvxjxtXdDvPtOfn2Gunxvz4fX/vaeI/bmli4rQZC5xyrXbYvzPWaonelxvbBlKbElYPlEzGgQc46Ssm2YXQT6622sSZ4naHZqr0Un/2MbfZ4bWcN2L/GxBL3FECAOBBQwkAgEe52Y/yjBrTnfT7/W42ceaO8CukbDjNrujw/lFPmbhlhttV8O52+5RWq3HXOXnB1TymbmgayFnprzRKZMNV9onTC279xMQDa88LKRnZ9Vuw+6f9k+6qOo3v/y6Qsl2Boc/G+qSpkpROTYuubhZRuZlj7FOR9eU7T0mUJ7q2Oxx2dvUNgVSluH62s+lAQbQ7lZYN7igBAPCgoQQAwIOGEgAAj7KdHvLrXCed86Z9PHjeBU85eVPudVf+j8THO+ua+KxRf3Xymjw0zcRt22xx3xhYzWP+VDtG2YIxylLJaNrYSd9z12gTn1ptq4lDV9XZkG83AT5jhj2PLx30olOuVaZdfSdjV6mqWqwCzXXkrqZ7El0FxNMqd6rc4dMuMXGX/e1G119/cbBTruoaJcXZWd99nuRf544x8blZ65y8XndNNPGH0s3E2WPiu/tMNPglAADAg4YSAACPsp0eot3b8lZ/s7fYh8253skr6LVRirNpbbaTbvaOjSt9bFd3aBzyiHrwk/WMOU7ev9cdZOLLTraL+n53e3wfj05F6W1amfjBCa84eW0y7XSOpXm2e7XXK4Occq2e+t3EdVbYqSO9X3b/RuacOMqWOzmkm/zxwMohUT56/txrp5i4EVMekILyN7q/s/udYdPBLQKay2SJxstP2BXXnnihmpP3xcF2tbRJ1wQ2b38zZNWfcjB1hDtKAAA8aCgBAPCgoQQAwKPcLGFX79mQPvBniy+3fww+K71uHSfdpZodK522o3kMPqHimn9vlomDY5IiIp/ttOPL/7z/JhM3e8E99+F28Wh1ubvM4bmTTjPxhA5vOXlHDLBLIO4/PLrxxUYPMC7psyqw2XqNpeV/75XqC3jmoKzlrbI7kGSd4ubdOvUYE3/Y9l0TH3HNDU65P7UNCcAdJQAAHjSUAAB4lJuu17KkG7oduKdV22bim7+2O1zkyI9lVqdU8eIRz4fNe/jmy01c54PSd6cs/LiFTbi9NXL1gPEmHje8riD2stNs1/ruGjauGufPTW9npxJcds2EiN/XdPQiE5f/juLYSa9d20nrPXa1pYLt28u6OsbHX3Ux8eMX2WGOs6//0in39bNVyqxO4XBHCQCABw0lAAAeFbLrdUXPOmHzMtZllmFNUk96YA2ktJDrsMrrd4cWL5VmL9qutFf6uAuwH111gYk/qJdj4vx162Nah1SXPSvwpOjJbl6WsovSH3mzXRVr9kvxrVPDF+0qTLfUnh+2XLvR7kpOLf6YGqZk6slo3MjE7d9b4eS9/54dXmoyOL5PdqvK9m9k6aBDnLzbe70bWrywTpXWhbzSqNhyZYk7SgAAPGgoAQDwoKEEAMCjQo5R7q6t910IUXll/VEm7tLgGydvyd9s3OLB9iYu+Pm3qD5L59ldBTbnuzsTtKtkrwHXnm3HKOuOjHxaytaLjjBxedxMtiw0HrPEJm4JX+7ganavidlyQMzrseg/dlztzYaPBXIqO+VGbrZj1a0eX+Dk5edVnEkhmw9raOL/1B/n5N119bcmPqTe35y8NqNCNrWPwKLza5k4t7a7Cft9Pd428QVZ7nhomtjNn4Pveuq+85xyNSXx3z3uKAEA8KChBADAo0J2vSJ+PvnsLzbRx+16nXHMcyZe+Z6dKvLo2u5OuY++7iKRGHvOEBOHLsA+fbe9Btzv1V9M7HYM+Z33j09MPGFMjRK8M3XowMotQze2cvJurm27Ni/OXmri+1/q5ZRr84hdPL0gZNP0cLadf7iTnn7Z4yauGpiWEuxqFREZd67t+s//I/zUkVRXfcVOEwc3phcR+Ue9mSaee85TTl7aOcHu0OBUL+WUC+Y574+wnIjI2sCi+ke/d6uJc952Nz8oDwNl3FECAOBBQwkAgAcNJQAAHoxRiki6stcLtWclsCIpoNWQhSb+4UJ3OcDDK+eauFGG3WPi0ZBpJI9e6KbDSRN7/IKQ0cePtna0eTt2SDRGzj7axE3k16iOkezyN2028ee93bEued+GwfHK+d1HOcVePsxOF/nvGPfR/6BLz/nCxjUfdfKqqmqhxUVE5IlXznTSjWaz2baIiHw/w4Rf3XKkk3XS3+1Y8//avuHkBZclDB1vDHKndthRxFe3ujsznZdllxvs8PEAJ6/pWHuM1h/8YOLyMCYZijtKAAA8aCgBAPCg61VE8rXttqs9e5unJPYlf81aE//nlHOdvLkD9jNxv+6fm3hgnehW5um79AQTT53gdgu2eG5pILVcotHk/IrZ3RpO3pKlTvq1oYHtRG4OhLXdFXEuz15t42uGR/hpblfri1samPid8443caPZPwj8Mj6f5r5gv3pyxuk3O1krL7abOk851k4dOW/uRU65de/bHT1UYNSjwavu9J/RnWzXeM4XP0Zc5/KGO0oAADxoKAEA8KDrVdynXhE7+fMWOulWA236C6keiLtG+Ql2Aecm4j7tWHGWv06c4ALzn7xYz8SfNevslJtzg30S8pjDbDf7N1PaSzhtR2x00gXzFptY584teWVRrCrjpzjpFuNtfJHYVY4yxO12PyAkvVd+SDrjiw2lql95QQsBAIAHDSUAAB40lAAAeDBGKSILc+2UkPRNdhWX0P52AMXTuXZaQf78RU5e65ttek3wdc+GvHz3UJ5wRwkAgAcNJQAAHhWy67XZPyY76QH/OCaQcqc0AAAqNu4oAQDwoKEEAMCDhhIAAA8aSgAAPGgoAQDwoKEEAMBDaa0TXQcAAMot7igBAPCgoQQAwIOGEgAADxpKAAA8krqhVEpppdR2pdT9EZbvq5TaVvS+VvGuH0qG85l6OKepJYrzObiovFZKJe3a4kn91KtSSotIa631gsBrI0TkeBFpLSJ/1Vq/GMn7kHih50UplSMiD4vIUSKSLiJTReQmrfVc3/tQfhRzTo8VkY9CilUXkfO01u+Eex/KhzC/uZ1F5DkRaScis0Wkr9b650B+MxFZLCKZWuu8Mq1wjCT1HWUYv4jIABH5KdEVQanVEpFxItJGROqLyBQReS+hNUKpaK2/1lpn7f2fiPQWkW0i8nGCq4YoKKUqSeF38hURqS0io0XkvaLXU0bKNZRa6ye11p+LyK5E1wWlo7WeorV+Tmu9QWudKyKPi0gbpVTdRNcNMXOFiLyttd6e6IogKt2kcLvGIVrr3VrrYSKiROTEhNYqxlKuoURKO05EVmut1ye6Iig9pVR1ETlPCu9CkJw6iMgM7Y7hzSh6PWXQUCIpKKUaiciTInJLouuCmDlHRNaJyKREVwRRyxKRzSGvbRaR7ATUJW5oKFHuKaX2E5FPROQprfXria4PYuYKEXlJJ/MThdgmIjVCXqshIlsTUJe4oaFEuaaUqi2FjeQ4rXVEj6Sj/FNKNZbC8a2XElwVlM4sEemolFKB1zoWvZ4yUq6hVEpVUkpVkcIB5UylVBWlVMr9d1YESqkaIjJBRL7VWt+Z6Pogpi4Xke+01gsTXRGUykQRyReRm5RSlZVSNxS9/kXiqhR7qdiAfCIiO6Vw7t2Iovi4hNYI0TpbRLqKyFVFk9D3/q9JoiuGUusjPMST9LTWe0TkLCk8n5tE5K8iclbR6ykj2RvK3SIyTSl1394XtNbdtNYq5H8TRUSUUlcppTYVva8gMVWGh3M+tdaji85f9eDcO631UhHOZ5L403dURERr3VZr/VxoYc5puVfcb+50rfUhWuuqWuu/aK2n781TSt0rhXPbd4tI0o5FJ/XKPAAAxFuy31ECABBXNJQAAHh4V3PvmXY+/bIJ8mnBW2rfpUqOc5o48TinnM/E4TuaesKdU+4oAQDwoKEEAMCDhhIAAA8aSgAAPGgoAQDwoKEEAMCDhhIAAA8aSgAAPGgoAQDwoKEEAMCDhhIAAA8aSgAAPGgoAQDw8O4ekmzmj/6Lief2GOnknXjDABNXG/tDmdUJACqK9A5tnPSSc+qa+NBeM528l5p+ZeJcnR/R8btf399JV313SkmrGBXuKAEA8KChBADAI6W6XkXbPTcLpMDJWtHdxq3HllWFEJTRvKmJl53d0MRbc/Kccm1yVph4fJtxJs55/zqnXKMJ9jqvxvTVTp7etsPE+X/8YWKV4f7Jr7zpMBPnVXXr2+SRafZ4u3cLgD/bcskRJj7tzolO3ti6v4Z9X66239/Q3+twnh4y1EkPmtvHxPmz50d0jGhwRwkAgAcNJQAAHqnV9erRst1KE6vKlZ08utXiY/XAo5z0j4OeMHGkXS3BUvN6P+Pm9Q5/jDe2Hmji5/92tolXHuv+yf96hduVE3T6xGtMrL79eV9VBVJWWpUqTnrhP7uYeNblw00c6fc6WjmZlZz07Jtr27zrQkvHDneUAAB40FACAOBBQwkAgEeFGaP8sO27Jj4zq6eTl88YZcykt2pu4tE3Px6SW/I/t7Hb9jfxuVnrIn7fhdmrbDzqKROnhVwbBkdUpu9289I37yq2XEWy5iY7zrzl0F2ekvGVWdlOIZp5zAthy/VueEhZVKdiUHa6XXBMUkTk18uHBVKlv99q/+aNYfN+u+CJsHkPnvCWiV84rLfNmBJ+Wko0uKMEAMCDhhIAAI8K0/WKsrGyl52W0a5S+OuwE3+90MTV76sRtlzmqk0mfq5BLSdvd137qPiAh95y8s7OWrvvyorIzD3axINuHeDkVZvJ4vnbj7ArHM0+fmTYcsEu7WinCER6jGDOK1saR/VZKF7BsbaLdVE/+/pvJw4rpvSfvb3tACf9j2/s1KzG49zfg6rv2QXNW8n3JlZdOrgHvSD85wW/58NaVDdxdozXSueOEgAADxpKAAA8aCgBAPBgjBIxdczl08LmrcrfaeI1v9Y3cfqp4Y9X/0c7Drnm0HT3s3rYR8AjHZMM9f6WziZmQ+8/az1gsYnPyT7byVt8ZRMT765tRw6VlqgU1Ntj4tk9ng1bru2Hdiy53e0LQnI3RvfhFVVgCohI6LjkiIgOcfrcM0xccM9+Tl7Otz9GX7dyhDtKAAA8aCgBAPCg6xUx9cGPnUz80OlfO3lNMrJMPPuS4RKRq2yYqdyu11ydH0i513zrAt28x751m4knnv+IU+6uerb7ttsF1zt5WW9+LxVd/qbNNhGMRaTxfctj+lnbLrAbAEsPN29Brl2Zp93DG2z9NtLVWlLBnUBCV9yJdBrID7szTaxPtButK1lRXPGkxx0lAAAeNJQAAHjQ9YqYyulvl8T4S92+Tt6vR79o4mhWb8kNeZpy3Ha7aevQxd2dvLSh9Uzc8kPbhXps9VuccnNOf9LEK3vmO3k5b5a4iiiFVb33hM0bvNwueJ0/b2FZVCdl6XYtTewubh5eu8+vddItR9jvb5qk/qbm3FECAOBBQwkAgAcNJQAAHoxRIm5aDHbHnLp1uD5MyejU+nG1iasuWhySG5ret4NzljlptvMuW/O7jzJxQcg1/LQprU3cStaXWZ1S0YruNU0cupF50NjtdUzceniumxnjjZF9gnX88xQxG2t3kaEY1wEAAIRFQwkAgAddr4ib/FlznXTWrNgeP2/fRf6kTU74lUN+neduApwjq8OURDwUiA7E7vShaBdah0hG40ZO+pRLJpvYN03rji/s5uo5U2K8E7LH8nvcdLCOoVPErlhil3Cq/cFvJnYnepUed5QAAHjQUAIA4JFaXa+B/pnQp7lCn5ZCxZHb4xATT2jj7rE3ObC4c5undjh59PbF184zDwt5Jfxepvl17FOXi16ze4ge0nSpU27ggZ/a94j7GOQ1z99g4sb//q4kVU1qG451u17/XX9s2LI9Z15g4na3zzFxrLsyQy15o6OJn+/8YsTvW/hMWxPX2jLZU7J0uKMEAMCDhhIAAA8aSgAAPFJrjDKwNEPoY8/Bx4pnP9DSycu5doMgtaRlZ5v4gRF2XDJ0rPqrbXaMQ0+P8fyVFJNef38nvfWo5ibeWcdec6edsy6i443uMCTklcphy8456ZmIjtn3954mnvZxeyev2WN2l4uS712TvNafsWPfhYosW17XxDlbSr66VbRu7/iJiQ+tHH5EtO/SE5x03Y8XmDie46jcUQIA4EFDCQCAR2p1vUaqUkXqeKkY0uvWcdLbXrMLP3epHH5lj+cnHW/i1vJDfCqXxHJPOtTE2fcscfLGthhu4uB0rMg35c7cd5EiwS7VP25pEr7g9zNM2ETcKSAV9Vt/V+ePnbRvIfScvj/GuzrGlo/sEFifGsGpQeHr99vzHZx03T/iNyUkiDtKAAA8aCgBAPCgoQQAwKNijlEi5Szr29ZJ/3jQ0GLL/XtdRyfd7vE1Jo5mN5JU9/up9idiQosJTt6rWxuaeFN+NRO/t7KTU27tlw2lOMP6Puuku1e1D/h3/eliJ69O73mB1CZ/peHI1+79UORjyKWXXss+K7DgmaZO3qyOL0RUp/Zv3mjiViPLZkwyFHeUAAB40FACAOBB1yuSRnC1HRGRNa82MPE7nR4OKV3JRMM22m7ZCQ8d65Squej72FUwBdWabVe7yvnwOiev3SDbHZq/abOJK8nvTrlGIem9frnE7Yo7rsr8qOuJ8iG4U4+ISP377Dkd2+S5kNLF36d9ttP9nrcZaVdOi/cuJuFwRwkAgAcNJQAAHnS9IqZWDzzKxJV6uItjP9r+TRMX6Miu0e5fcpqJBzd/18kLrrgT7GoN9eWFdnWZmrPoai2JeiMmB2I3L57dYJmv1Nl3IcTc+quPNHHdUZE9YTrvBdvd2rTheidvZJPPGysDhQAAA2tJREFUS1yHGz+6wkm3/i3xK2ZxRwkAgAcNJQAAHjSUAAB4MEaJUtl64RFO+sdBT4QtG9w0OVfnRnT8D9vaccnQTZeDO4FsLtjl5HV/ZJCJD5jl7iKBxAlu/twgs/hpIyIiGbsq6l4fsffQjJOcdJ9jXghTUqR9X7t5+Y8H2ucN+l30oVPu+loLTZyp7IbYuTp05Dr8vVjw+5wz+gYTt/57Ylbf8eGOEgAADxpKAAA8kr7rVWXY/4TK1fcksCYV06qe7lLivsWNg12l0SzMHLrpcvAY/7e6u5PX8JM/TJyo1TzwZ1uPam7is7M+CMnluj0eGo1wN8ie3NV2eR5e2R0CcaZzXBd+akfw2xvp9zq4QpaIyMjxtku4xT9/MnHI17xc4C8TAAAPGkoAADxoKAEA8Ej6Mcq05k1M/PNRz4ctF5w+0ODDpP/PTqj0unZ5sYsPmZLAmliPN/jaSX85PsvETxzTzcR5q9cIyoe0kOv04Hc0Yxsjy7GS8fk0J33b4P4m/vqBYTH9rOV5u530Q2t6mnjZlY2dvOa/2Wkg5XFcMog7SgAAPGgoAQDwSP4+yA2bTHjwSzeZ+NDj5jjFlj/c2sRZ7yZ+Nfpkltvebrh77/4TYn78U347z8RrJjW0Gcotd+eldjeSC7NXOXknVN1m4icqh99ZBIkTOpXgpc0Hmzjzs2mhxREj9T62q+p0aXyzkze9/9BSHfvsobc76QMfC66KNU+SFXeUAAB40FACAOCR9F2v+es3mLh5YDHd9SHlqkr5eDozFWSust3dx0y/1Mn7psurYd+3Kn+niXu+bBctbzViuVOu8krbjdo4N/zC2WOe6WLiN6od6eRtOaSBibO3JG+XT0UycvbRJm4ivyawJqktf81aEzf+91on74x/dy3VsQ+U1NyAgDtKAAA8aCgBAPCgoQQAwCPpxyhR9vIXLDZxnd5u3hkS2RhHM7HjyXmect56/PFH2Lxqvy+z5aI8PmJvRY/weVkfZoXPBBKIO0oAADxoKAEA8KDrFUCZSdtll1d6dP1BTl6dFyaHFgfKBe4oAQDwoKEEAMCDhhIAAA/GKAGUmZa3fm/iSVI1gTUBIscdJQAAHjSUAAB4KK11ousAAEC5xR0lAAAeNJQAAHjQUAIA4EFDCQCABw0lAAAeNJQAAHj8P0ZzyAz8BHBFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x576 with 16 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 8))\n",
    "\n",
    "for idx in range(16):\n",
    "    plt.subplot(4, 4, idx + 1)\n",
    "    plt.axis('off')\n",
    "    plt.title('[{}]'.format(np.argmax(mnist.train.labels[idx])))\n",
    "    plt.imshow(mnist.train.images[idx].reshape((28, 28)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，定义用于训练的网络，首先定义网络的输入。\n",
    "\n",
    "这里我们直接使用上面的数据作为输入，所以定义两个placeholder分别用于图像和lable数据\n",
    "\n",
    "为了让网络更高效的运行，多个数据会被组织成一个batch送入网络，两个placeholder的第一个维度就是batchsize，因为我们这里还没有确定batchsize，所以第一个维度留空。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.706044Z",
     "start_time": "2018-06-01T06:32:51.698913Z"
    }
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, [None, 784], name='x')\n",
    "y = tf.placeholder(tf.float32, [None, 10], name='y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为我们输入的是图片展开后的一维向量，所以第一步就需要先把一维向量还原为二维的图片。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.719298Z",
     "start_time": "2018-06-01T06:32:51.707730Z"
    }
   },
   "outputs": [],
   "source": [
    "x_image = tf.reshape(x, [-1, 28, 28, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:07:36.530623Z",
     "start_time": "2018-06-01T06:07:36.522665Z"
    }
   },
   "source": [
    "接下来，我们定义第一个卷积层：<br/>\n",
    "使用32个5X5的卷积核对输入数据进行卷积；<br/>\n",
    "padding方式选择SAME，所以输出数据的宽高为28x28,但是深度已经从原来的1变成了32；<br/>\n",
    "本层卷积的激活函数为relu。<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.764292Z",
     "start_time": "2018-06-01T06:32:51.721295Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:\n",
      "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "  * https://github.com/tensorflow/io (for I/O related ops)\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "WARNING:tensorflow:From /home/silence/miniconda3/lib/python3.7/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `layer.__call__` method instead.\n"
     ]
    }
   ],
   "source": [
    "with tf.name_scope('conv1'):\n",
    "    C1 = tf.contrib.slim.conv2d(\n",
    "        x_image, 32, [5, 5], padding='SAME', activation_fn=tf.nn.relu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来进行第一个池化：<br/>\n",
    "窗口为2x2，stride为2的最大池化；<br/>\n",
    "池化后输出变成了14x14，深度32。<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.774112Z",
     "start_time": "2018-06-01T06:32:51.766784Z"
    }
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('pool1'):\n",
    "    S2 = tf.contrib.slim.max_pool2d(C1, [2, 2], stride=[2, 2], padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:10:16.678485Z",
     "start_time": "2018-06-01T06:10:16.671472Z"
    }
   },
   "source": [
    "接下来，我们定义第二个卷积层：<br/>\n",
    "使用64个5X5的卷积核对输入数据进行卷积：<br/>\n",
    "padding方式选择SAME，输出14x14,深度为64：<br/>\n",
    "本层卷积的激活函数为relu。<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.805912Z",
     "start_time": "2018-06-01T06:32:51.776959Z"
    }
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('conv2'):\n",
    "    C3 = tf.contrib.slim.conv2d(\n",
    "        S2, 64, [5, 5], padding='SAME', activation_fn=tf.nn.relu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来进行第二个池化：<br/>\n",
    "窗口为2x2，stride为2的最大池化；<br/>\n",
    "池化后输出变成了7x7，深度64。<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.814748Z",
     "start_time": "2018-06-01T06:32:51.807560Z"
    }
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('pool2'):\n",
    "    S4 = tf.contrib.slim.max_pool2d(C3, [2, 2], stride=[2, 2], padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "池化后的数据是3维的，这里做一个拉平的操作，将3维数据展开到1维，然后送入两层全连接，全连接隐层中神经元个数分别为1024，10。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.856740Z",
     "start_time": "2018-06-01T06:32:51.817287Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/silence/miniconda3/lib/python3.7/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.flatten instead.\n"
     ]
    }
   ],
   "source": [
    "with tf.name_scope('fc1'):\n",
    "    S4_flat = tf.contrib.slim.flatten(S4)\n",
    "    C5 = tf.contrib.slim.fully_connected(\n",
    "        S4_flat, 1024, activation_fn=tf.nn.relu)\n",
    "\n",
    "#with tf.name_scope('fc2'):\n",
    "#    F6 = tf.contrib.slim.fully_connected(C5, 84, activation_fn=tf.nn.relu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:17:39.880958Z",
     "start_time": "2018-06-01T06:17:39.869797Z"
    }
   },
   "source": [
    "对特征添加一个dropout，以预定的概率丢弃特征中的某些数据，这样可以提高网络的推广能力，减少过拟合的可能性。\n",
    "\n",
    "需要注意的是，dropout仅在训练的时候使用，验证的时候，需要关闭dropout，所以验证时候的keep_prob是0。\n",
    "\n",
    "dropout的输出最终送入一个隐层为10的全连接层，这个全连接层即为最后的分类器。<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:51.913419Z",
     "start_time": "2018-06-01T06:32:51.860766Z"
    }
   },
   "outputs": [],
   "source": [
    "with tf.name_scope('dropout'):\n",
    "    keep_prob = tf.placeholder(name='keep_prob', dtype=tf.float32)\n",
    "    C5_drop = tf.nn.dropout(C5, rate = keep_prob)\n",
    "\n",
    "with tf.name_scope('fc3'):\n",
    "    logits = tf.contrib.slim.fully_connected(C5_drop, 10, activation_fn=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来定义loss和用于优化网络的优化器。loss计算使用了sparse_softmax_cross_entropy_with_logits,\n",
    "\n",
    "这样做的好处是labels可以不用手动做one_hot省了一些麻烦。这里使用了Adam优化器，学习率为0.0001。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:52.084738Z",
     "start_time": "2018-06-01T06:32:51.915376Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-18-cff470ff0f59>:2: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "\n",
      "Future major versions of TensorFlow will allow gradients to flow\n",
      "into the labels input on backprop by default.\n",
      "\n",
      "See `tf.nn.softmax_cross_entropy_with_logits_v2`.\n",
      "\n",
      "Conv/weights:0\n",
      "INFO:tensorflow:Summary name Conv/weights:0 is illegal; using Conv/weights_0 instead.\n",
      "Conv/biases:0\n",
      "INFO:tensorflow:Summary name Conv/biases:0 is illegal; using Conv/biases_0 instead.\n",
      "Conv_1/weights:0\n",
      "INFO:tensorflow:Summary name Conv_1/weights:0 is illegal; using Conv_1/weights_0 instead.\n",
      "Conv_1/biases:0\n",
      "INFO:tensorflow:Summary name Conv_1/biases:0 is illegal; using Conv_1/biases_0 instead.\n",
      "fully_connected/weights:0\n",
      "INFO:tensorflow:Summary name fully_connected/weights:0 is illegal; using fully_connected/weights_0 instead.\n",
      "fully_connected/biases:0\n",
      "INFO:tensorflow:Summary name fully_connected/biases:0 is illegal; using fully_connected/biases_0 instead.\n",
      "fully_connected_1/weights:0\n",
      "INFO:tensorflow:Summary name fully_connected_1/weights:0 is illegal; using fully_connected_1/weights_0 instead.\n",
      "fully_connected_1/biases:0\n",
      "INFO:tensorflow:Summary name fully_connected_1/biases:0 is illegal; using fully_connected_1/biases_0 instead.\n"
     ]
    }
   ],
   "source": [
    "cross_entropy_loss = tf.reduce_mean(\n",
    "    tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))\n",
    "\n",
    "#l2_loss = tf.add_n([\n",
    "#    tf.nn.l2_loss(w)\n",
    "#    for w in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)\n",
    "#])\n",
    "\n",
    "for w in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES):\n",
    "    print(w.name)\n",
    "    tf.summary.histogram(w.name, w)\n",
    "\n",
    "total_loss = cross_entropy_loss\n",
    "#total_loss = cross_entropy_loss + 7e-5 * l2_loss\n",
    "tf.summary.scalar('cross_entropy_loss', cross_entropy_loss)\n",
    "#tf.summary.scalar('l2_loss', l2_loss)\n",
    "tf.summary.scalar('total_loss', total_loss)\n",
    "\n",
    "#optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.3).minimize(total_loss)\n",
    "optimizer = tf.train.AdamOptimizer(1e-4).minimize(total_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:25:56.449132Z",
     "start_time": "2018-06-01T06:25:56.438340Z"
    }
   },
   "source": [
    "需要注意的是，上面的网络，最后输出的是未经softmax的原始logits，而不是概率分布，\n",
    "要想看到概率分布，还需要做一下softmax。\n",
    "\n",
    "将输出的结果与正确结果进行对比，即可得到我们的网络输出结果的准确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:39:50.010829Z",
     "start_time": "2018-06-01T06:39:49.997501Z"
    }
   },
   "outputs": [],
   "source": [
    "pred = tf.nn.softmax(logits)\n",
    "correct_pred = tf.equal(tf.argmax(y, 1), tf.argmax(logits, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "saver用于保存或恢复训练的模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:32:52.127795Z",
     "start_time": "2018-06-01T06:32:52.103115Z"
    }
   },
   "outputs": [],
   "source": [
    "batch_size = 100\n",
    "trainig_step = 300\n",
    "\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上定义的所有操作，均为计算图，也就是仅仅是定义了网络的结构，实际需要运行的话，还需要创建一个session，并将数据填入网络中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-01T06:35:22.270272Z",
     "start_time": "2018-06-01T06:33:18.829198Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNCTIONS READY\n",
      "After 1 training epoch, Validation accuracy : 0.9406\n",
      "After 2 training epoch, Validation accuracy : 0.9628\n",
      "After 3 training epoch, Validation accuracy : 0.9764\n",
      "After 4 training epoch, Validation accuracy : 0.9806\n",
      "After 5 training epoch, Validation accuracy : 0.9804\n",
      "After 6 training epoch, Validation accuracy : 0.9848\n",
      "WARNING:tensorflow:From /home/silence/miniconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to delete files with this prefix.\n",
      "After 7 training epoch, Validation accuracy : 0.986\n",
      "After 8 training epoch, Validation accuracy : 0.9836\n",
      "After 9 training epoch, Validation accuracy : 0.9858\n",
      "After 10 training epoch, Validation accuracy : 0.9878\n",
      "After 11 training epoch, Validation accuracy : 0.9892\n",
      "After 12 training epoch, Validation accuracy : 0.988\n",
      "After 13 training epoch, Validation accuracy : 0.9898\n",
      "After 14 training epoch, Validation accuracy : 0.99\n",
      "After 15 training epoch, Validation accuracy : 0.9898\n",
      "After 16 training epoch, Validation accuracy : 0.9912\n",
      "After 17 training epoch, Validation accuracy : 0.9914\n",
      "After 18 training epoch, Validation accuracy : 0.9908\n",
      "After 19 training epoch, Validation accuracy : 0.9912\n",
      "After 20 training epoch, Validation accuracy : 0.9916\n",
      "After 21 training epoch, Validation accuracy : 0.9922\n",
      "After 22 training epoch, Validation accuracy : 0.9922\n",
      "After 23 training epoch, Validation accuracy : 0.9914\n",
      "After 24 training epoch, Validation accuracy : 0.9926\n",
      "After 25 training epoch, Validation accuracy : 0.9924\n",
      "After 26 training epoch, Validation accuracy : 0.9934\n",
      "After 27 training epoch, Validation accuracy : 0.9906\n",
      "After 28 training epoch, Validation accuracy : 0.9914\n",
      "After 29 training epoch, Validation accuracy : 0.9918\n",
      "After 30 training epoch, Validation accuracy : 0.9932\n",
      "The training is finish!\n",
      "The test accuarcy is: 0.992\n"
     ]
    }
   ],
   "source": [
    "merged = tf.summary.merge_all()\n",
    "with tf.Session() as sess:\n",
    "\n",
    "    writer = tf.summary.FileWriter(\"logs/\", sess.graph)\n",
    "\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    print(\"FUNCTIONS READY\")\n",
    "\n",
    "    #定义验证集与测试集\n",
    "    validate_data = {x: mnist.validation.images, y: mnist.validation.labels, keep_prob: 0}\n",
    "    test_data = {x: mnist.test.images, y: mnist.test.labels, keep_prob: 0}\n",
    "    for epoch in range(30):\n",
    "        for i in range(trainig_step):\n",
    "            xs, ys = mnist.train.next_batch(batch_size)\n",
    "            _, loss, rs = sess.run(\n",
    "                [optimizer, cross_entropy_loss, merged],\n",
    "                feed_dict={ x: xs, y: ys,keep_prob: 0.5})\n",
    "            writer.add_summary(rs, i)\n",
    "\n",
    "        #每1次epoch打印一次验证准确率\n",
    "        validate_accuracy = sess.run(accuracy, feed_dict=validate_data)\n",
    "        print(\"After %d training epoch, Validation accuracy : %g\"% (epoch+1, validate_accuracy))\n",
    "        saver.save(sess, './model.ckpt', global_step=epoch)\n",
    "\n",
    "    print(\"The training is finish!\")\n",
    "    #最终的测试准确率\n",
    "    test_accuracy = sess.run(accuracy, feed_dict=test_data)\n",
    "    print(\"The test accuarcy is:\", test_accuracy)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
